Variability system and analytics for continuous reliability in cloud-based workflows

ABSTRACT

The present disclosure relates to automatically detecting software defects and determining the source of the defect. An example method generally includes receiving an output data set from a host server executing a test operation and comparing the output data set to a performance model. A test system identifies that a defect exists in software executing on the host server based on determining that the output data set deviates from the performance model by more than a threshold. The test system retrieves a source code update from a source code repository that was committed to the source code repository before the execution of the test operation. The test system compares a scope of the source code update to the defect and notifies a development team related to the source code update of the defect upon determining that the scope of the source code update and the defect match.

BACKGROUND Field

Embodiments presented herein generally relate to quality control testingand defect detection in software applications, and more specifically toperforming quality control testing concurrently on multiple softwaredeployments.

Description of the Related Art

Commercial software is increasingly being provided using distributedarchitectures through public and private cloud computing environments toglobal customers. These software products can includes commonprogramming subsystems and individualized program components forspecific customers, markets, regions, etc. Commonly, developers work inteams devoted the creating and maintaining a single subsystem orindividualized component. Thus, many development teams work in parallelto maintain the program, but with minimal inter-team communication. Aproblem with such distributed independent development teams is thatdefects can be introduced into the program when the individual teamseach update the portion of the source code they are responsible forupdating, maintaining, and developing.

The defects introduced by the independent updates often are notdiscovered until they degrade the performance of the productionsoftware, creating issues for customers and negatively impacting thecustomer experience. After the defect is discovered, developers need tolocate the defect in one of the prior software updates before acorrection can be made. Reviewing the numerous independent source coderevisions is very labor intensive, requiring either a large number ofdevelopers to be devoted to the project or allowing the defect—with theassociated customer issue—to persist for an extended period of time.

SUMMARY

One embodiment of the present disclosure includes a method for detectingand sourcing software defects. The method includes receiving an outputdata set from a host server executing a test operation in a softwaresystem. A test system compares the output data set to a performancemodel and identifies that a defect exists in software executing on thehost server based, at least in part, on determining that the output dataset deviates from the performance model by more than a threshold. Thetest system retrieves a source code update from a source coderepository, wherein the source code update was committed to the sourcecode repository before the execution of the test operation and comparesa scope of the source code update to the defect. Upon determining thatthe scope of the source code update and the defect match, the testsystem notifies a development team related to the source code updatethat an error exists in the source code update.

Another embodiment provides a computer-readable storage medium havinginstructions, which, when executed on a processor, operates to detectand source software defects. The operation includes receiving an outputdata set from a host server executing a test operation in a softwaresystem. A test system compares the output data set to a performancemodel and identifies that a defect exists in software executing on thehost server based, at least in part, on determining that the output dataset deviates from the performance model by more than a threshold. Thetest system retrieves a source code update from a source coderepository, wherein the source code update was committed to the sourcecode repository before the execution of the test operation and comparesa scope of the source code update to the defect. Upon determining thatthe scope of the source code update and the defect match, the testsystem notifies a development team related to the source code updatethat an error exists in the source code update.

Still another embodiment of the present invention includes a processorand a memory storing a program, which, when executed on the processor,performs an operation for detecting and sourcing software defects. Theoperation includes receiving an output data set from a host serverexecuting a test operation in a software system. A test system comparesthe output data set to a performance model and identifies that a defectexists in software executing on the host server based, at least in part,on determining that the output data set deviates from the performancemodel by more than a threshold. The test system retrieves a source codeupdate from a source code repository, wherein the source code update wascommitted to the source code repository before the execution of the testoperation and compares a scope of the source code update to the defect.Upon determining that the scope of the source code update and the defectmatch, the test system notifies a development team related to the sourcecode update that an error exists in the source code update.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentdisclosure can be understood in detail, a more particular description ofthe disclosure, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlyexemplary embodiments and are therefore not to be considered limiting ofits scope, may admit to other equally effective embodiments.

FIG. 1 illustrates an example system, according to an embodiment.

FIG. 2 is a flow chart illustrating a process for generating andprocessing simulation cases to document system performance, according toan embodiment.

FIG. 3 is graph illustrating a software defect, according to anembodiment.

FIG. 4 is flow chart illustrating a process for identifying the sourceof a software defect, according to an embodiment.

FIG. 5 illustrates an example computing system for identifying andsourcing software defects, according to an embodiment.

DETAILED DESCRIPTION

Generally, developers of commercial software applications can utilize apublic cloud architecture to deliver continuous global access tosoftware products. Applications deployed via a public cloud architectureoften include many subsystems that work together to provide the servicesto users. In one example, a cloud-based application can serve separateuser interfaces to users based, for example, on a location of the user.Each of these interfaces may implement region-specific variations onvarious features. For example, while generating invoices for purchaseorders may use similar inputs, the generated invoices may differ basedon tax regulations in different geographical regions. To allow users togenerate invoices that comply with regional regulations, each regionalinterface can provide application functionality using region-specificprocessing rules and common subsystems that need not be region-specific.

Commonly, the individual components, or modules, of the software systemare developed and maintained by separate teams that work concurrentlyand often with minimal inter-team coordination. These largelyindependent development teams allow developers to quickly add featuresor modify existing features in the software modules. However, the lackof centralized control and coordination between teams responsible fordifferent components in a software system greatly increases theprobability of a defect being released into the live software systemthat adversely affects customer reliability, performance, or both, asindependent teams may not be able to fully test updates to individualsoftware system components to identify problems that may result frominteractions between existing software modules and the updated softwaremodule.

Software defects introduced by various development teams are generallydifficult and time consuming to repair. First, the defect itself may notbe immediately detected by developers, meaning that additional softwareupdates to other or the same module can be committed to a coderepository and deployed on a publically-available version of thesoftware system before the original defect is detected. Once developershave discovered a defect in a previous version of a module, the task ofmanually evaluating the code to perform a root cause analysis is verytime consuming, and implementing a repair can require close coordinationbetween multiple development teams. Until a repair can be developed,tested, and deployed, the introduction of defects in a software systemmay negatively impact customers due to the degraded performance andreliability of the product. Further, because development teams may bereallocated from product development to debugging tasks, repairingdefects in software systems may negatively impact product developmentdue to the significant engineering resources that need to be diverted toperform a root cause analysis of the defect introduced into the softwaresystem and repair the identified source(s) of the defect.

Embodiments presented herein provide a system for detecting softwaredefects in near-real time and automatically mapping the defects to theportion of the code base responsible for introducing the software defectinto an application. The system maintains a performance baseline byperiodically executing a set of test cases within the system andrecording the performance information at each time period as well ascontextual metadata documenting the state of the software system.Deviations from previous system performance may serve as a proxy for theidentification of defects introduced into the software system, as it maybe inferred that a large negative performance deviation from priorperformance that coincides with one or more source code updates isindicative of a problem with the one or more source code updates.

FIG. 1 illustrates an example system 100, according to one embodiment.As shown, system 100 includes a controller 110 and one or more hostservers 120, a datastore 125 that includes a source code repository 130,and a database 140, each of which communicate with each other vianetwork 150.

Controller 110 periodically tests software applications deployed on oneor more host servers 120 to identify the performance of the softwareapplications and detect defects in the software applications based ondeviations between prior performance and current performance of a testoperation. To test a software application, controller 110 may beconfigured with different sets of test operations that controller caninvoke on the one or more host servers 120. Each of the test operationsmay include, for example, information defining messages to betransmitted to the one or more host servers 120 to invoke an operation,parameters to include in the messages invoking the operation, and thelike. In some cases, a definition of the test operations may include asequence of operations that are to be performed by the one or more hostservers 120. Each operation in the sequence of operations may beassociated with a weighting indicating an importance of the operationand a maximum amount of time for the performance of the operation beforecontroller 110 determines that the operation failed.

When controller 110 receives performance data from the one or more hostservers 120, controller 110 can compare the received performance data tohistorical performance data and/or performance threshold information ina definition of a test operation to determine if the tested softwareapplication deployed on the one or more host servers 120 is workingproperly. If the received performance data deviates from historicalperformance data by an amount less than a threshold amount, or if thereceived performance data indicates that the completion time of the testoperation complies with the threshold time defined for the operation,controller 110 can determine that the software application executing onthe one or more host servers 120 is working properly. If, however,controller 110 determines that the received performance data deviatesfrom historical performance data by an amount greater than the thresholdamount of that the received performance data does not comply with thethreshold time defined for the operation, controller 110 can determinethat a problem exists in the software application deployed on the one ormore host servers 120.

In some cases, when controller 110 detects that an execution time forone or more test operations exceeds a threshold time, controller 110 canexamine an execution time trend to determine whether the failure toexecute the test operation within the threshold time is a temporaryfailure or a persistent failure. A temporary failure, which may resultin a number of testing failures followed by a number of successfulexecutions of the test operation, may indicate, for example, a hardwarefailure causing one or more host servers 120 to temporarily go offline.In contrast, a persistent failure, which may result in a continuousnumber of testing failures, may indicate a code error introduced intothe software application components executing on one or more hostservers 120. If controller 110 detects a persistent failure in one ormore components of a software application executing on one or more hostservers 120, controller 110 can examine the historical performance dataof the test operations to identify a time at which the problemsinitially appeared in the performance data of the one or more hostservers 120. Based on the initial appearance time of performanceproblems in the software application, controller 110 can examine a coderepository (e.g., source code repository 130) for code commits thatoccurred before the initial appearance of a problem in committed anddeployed source code (e.g., the code commit that occurred immediatelybefore the initial appearance of the problem).

Upon identifying the one or more code commits that occurred around thetime at which degraded performance at the host servers 120 initiallyoccurred, controller 110 can identify the one or more developmentresources responsible for the code commit that is likely to be the causeof the problems experienced during testing. A development resource mayinclude a development team having a designated contact person, anautomated troubleshooter, or other development resources that canrectify errors detected in program source code. In some cases,controller 110 can generate an e-mail or other electronic message to besent to a designated contact person for the code commit that wasdetermined to be the likely cause of the degraded performance at the oneor more host servers 120. In some cases, controller 110 can identify theprevious working version of the software in source code repository 130(e.g., the source code associated with the code commit that occurredimmediately before the source code commit that was determined to be thecause of the problem). Controller 110 can subsequently roll back thesoftware deployed on the one or more host servers 120 to the previousworking version until the debugging team fixes the error and commits apatched version of the software to source code repository 130.

The one or more host servers 120 provide software to a global customerbase by executing a number of software modules that function together toprovide a complex set of software features for numerous region specificinstances of the software. For example, host servers 120 can providedistinct user interfaces for specific countries, such as distinct userinterfaces for France, Spain, and the United States. Eachcountry-specific interface can have a dedicated software module forapplying country specific rules, such as accounting practices,performing tax calculations, and the like, as well as rely on commonsoftware modules, such as communicating with third party application oraccessing databases that store system-wide information. As discussedabove, the combination of dedicated and common software modules cancause a change in one module to affect many different customer groups.

Source code repository 130 stores the source code files that constitutethe software modules being executed by host servers 120, and tracks andstores different versions of the source code files as they are updatedby developers. That is, each code module versioned as it is stored inthe code repository 130, and each time a development team commits anupdate to the code repository 130, the version for the code module isincremented and the date and time of the update are recorded. Database140 stores performance results and contextual metadata created bycontroller 110 during operation of the system 100.

FIG. 2 is a flowchart illustrating operations 200 that may be performedby a controller 110 to test software modules deployed on one or morehost servers 120 and evaluate the performance of the software modulesdeployed on the one or more host servers 120, according to anembodiment.

As illustrated, operations 200 may begin at step 210, where controller110 creates a new instance of a test operation to be performed on theone or more host servers 120. The test operation may be generated usinga test plan that includes a test operation identifier, a software buildnumber, a test case context, and an execution plan. The test operationidentifier may identify the specific test operations being performed.The test case context generally describes how test cases are to bedivided and processed. For example, the test case context may specifythat the execution of the operations specified in the test operation areto be executed in parallel and distributed across host servers 120 thatprovide the software application to customers in different geographicregions.

At step 220, controller 110 generates the set of simulation test casesfor the test case context, including a session ID that uniquelyidentifies the specific test set being run. At step 230, controller 110adds a production software build ID to each simulation case that recordsthe version of the software that the simulation case will be executedwithin.

At step 240, controller 110 initializes each test case on a host server120 according to the test case context. Each host server 120 runs thetest cases assigned to it, and produces a result data set. Theindividual test cases from the set of test cases can be executed acrossa group of host servers 120 or on a single host server 120, depending onthe test case context.

At step 250, controller 110 receives the result data set from each hostserver 120 that executed a test simulation, and processes the results togenerate a score for each simulation and to aggregate the results intoan output data set.

The output data set includes metadata describing the context of the testoperation, the score, statistics related to the test operation, and timeseries statistics describing how long the test simulations took toexecute. The time series information for each test simulation isgenerated by the host server 120 that executes the simulation, andcontroller 110 aggregates the individual time series data together forthe test operation. The output data set for each test operation isstored in database 140 by controller 110. Controller 110 periodicallyexecutes the number of test operations, each of which generate an outputdata set, and analyzes the collection of output data sets in database140 to provide a near-real time visualization of the softwareperformance.

FIG. 3 illustrates an example visualization of system performance overtime generated by controller 110, according to an embodiment. Thevisualization of system performance may include a plurality of graphs,with each graph representing a performance metric for a particularvariation of the software system deployed on a host server 120. Forexample, as illustrated in FIG. 3, the visualization generated bycontroller 110 may include a first graph 300 illustrating the set-uptime for US based Companies and a second graph illustrating the set-uptime for United Kingdom-based companies. Controller 110 creates thefirst graph by retrieving output data sets for a test operation designedto set up a company in a United States-specific version of the softwareapplication. Each point in the first graph represents the set-up timefrom a different output data set, arranged as a timeline. As shown, theset-up time increased substantially for point 320 as compared to points310 and 330. FIG. 3 also includes a second graph 340 illustrating theset-up times for United Kingdom based companies created by retrievingoutput data sets for a test operation designed to set up a number ofcompanies in the United Kingdom-specific version of the softwareapplication. Graph 340 and 300 represent approximately the same timeperiod. The visualization generated by controller 110 can includeadditional test operation results in a stacked or layered manner todepict the overall time taken for the combined test operations as wellas the individual contributions for each test operation.

Controller 110 can also generate, as a separate visualization oraccompanying the graph in FIG. 3, a visual indicator of the score ofindividual software modules or regional instances. For example, the USand UK regions graphs 300, 340 in FIG. 3 could have each point depictedas a real-time representation of the state of that system, where theresults of the most recently run test operations are indicated with acolor, value, or both representing the state. A state representation forthe time just after point 310 would indicate both the US and UK regionsoperating as expected using, for example, a block of green color next tothe region. After the tests run at point 320 the indicator for the USregion would turn red, indicating that the controller 110 had detecteddegradation in system performance indicative of a problem with thesoftware.

The output data set for each test operation uses a structured dataformat for all test operations to unify the data. The structured dataincludes, for example, metadata describing the software build/versionthat the test was run on, metadata defining the use-case that wastested, and monitoring metrics. The use-case metadata, for instance,includes the geographic region being tested, the number and type ofoperations executed, such as how many records are created/modified andeach step of the process. The monitoring metrics include the responsetime for each operation and error rates. Controller 110 uses theinformation in the output data set to derive a score for the individualtest operations, which may be used to identify degraded performance in asoftware system and the one or more source code modules that are likelyto have caused the degraded performance.

Controller 110 derives the score for individual test operations bycomparing the performance of the current test operation to theperformance of prior test operations. For instance, controller 110 canassign a score based at least in part on predetermined performancethresholds identified by developers, i.e., 10-20 milliseconds scoreshighest, 20-30 milliseconds is acceptable, and 40+ milliseconds is afailure. Controller 110 can also assign a score by applying astatistical analysis of the test operation, such as, for example,comparing the test operation to the average or median times for prioroperations. Controller 110 can apply a polynomial regression to a set ofearly test operation outputs to establish a performance model and scorethe current test operation using the deviation from the performancemodel. Similarly, controller 110 can use running averages and standarddeviations to score a test operation result. Controller 110 can combinethese individual techniques as well as use them with rules definingpriority or relative weight, or both. For example, a test operation withthree steps—create a company entity, assigning the company anot-for-profit tax status, and linking the entity to a third partypayroll provider—could have rules for scoring the operation thatemphasize particular steps in the test operation. For instance, a rulecould assign a low score because the initial step of creating thecompany entity fell outside acceptable time thresholds, regardless ofwhat happened in the following steps. In another example, a rule mayspecify that the first step be assigned ⅔ of the score weight, while theremaining two steps account for only ⅓ of the score. Thus, developerscan tailor the score calculation to give an accurate reflection ofsystem performance.

After controller 110 has detected a performance change representing adefect in the code operating on the host servers 120, controller 110automatically performs a root cause analysis (RCA) to identify thesource of the defect within the source code. FIG. 4 illustrates aprocess for identifying the cause of a defect, according to anembodiment. As shown, controller 110 begins the process 400 at step 410by retrieving from database 140 the output data set identified duringanalysis of the output data sets as departing from expected operationsand the output data set predating the identified output data set. Atstep 420, controller 110 accesses the context data of the output dataset to determine the version of the software running at the time theoutput data set was created. At step 430, controller 110 retrieves thesource code updates logged into source code repository 130 between thetime that the identified output data set was created and the time theprior output data set was created. This window may be defined as thetime period when a new software update responsible for the defect wasadded to the production code. After controller 110 has received the oneor more software updates, at step 440, controller 110 determines whichsoftware update is the source of the defect by comparing the scope ofthe defect to the responsibilities for each development group thatsubmitted a code update.

For example, the output data set for point 320 in FIG. 3 indicated adefect in the US region for the process of creating companies, but nosimilar defect was detected in the same process for the UK region.Controller 110, therefore, compares the responsibilities of thedevelopment groups who made a software update in the relevant timeperiod with the scope of the defect. That is, controller 110 determineswhether the software modules that received an update in the relevanttime period are executing on the host server 120 identified as beingaffected by the code defect. If so, the code module that is bothexecuting on the identified host server 120 and received a code updateis identified as responsible for the defect. Source code repository 130includes metadata identifying the individual developer who commits eachupdate to the code, and controller 110 uses the metadata to alert theresponsible person.

In the example, the defect would be linked to an update from the USdevelopment group because the defect was limited to the US host server120, not the UK development group. If the UK had also experienced adefect, then controller 110 would seek a software module formdevelopment group with a broader scope, such as a module responsible formaintaining a system wide company profile database. At step 450,controller 110 notifies the development group identified of the defectand the associated code update. In cases where only a single code updatehas been logged in the time between the two output data sets, controller110 assigns the defect to the development team responsible for thatsingle code update without needing to evaluate the scope of developmentteam responsibility. As discussed above, controller 110 may, in somecases, assign the defect to another development resource, such as anautomated troubleshooter, to rectify the detected defect.

Thus, controller 110 automatically detects and attributes code defectsin near real time using the changes in system performance. The frequencythat the controller 110 runs each test operation impacts both howquickly a defect is discovered and the potential number of source codeupdates that need to be evaluated. Controller 110, therefore, can varythe frequency of each test operation to prioritize some softwaresubsystems over other subsystems. For example, critical subsystems thatprovide services to many or all customer groups/regions can have testoperations run more frequently than other subsystems that providenon-essential services. As another example, each development team couldset a relatively high frequency for test operations for the team'ssubsystem for a limited time before and after introducing a new softwareupdate. The increased frequency would allow controller 110 to quicklyidentify defects and notify the development team. Thus, the developmentteam would be proactively debugging the update by reducing the time tofix any potential defects. After the software update had executed for asatisfactory amount of time, the development team could reduce the testoperation frequency. In an embodiment, controller 110 includes a set oftest operations from each development team that is periodically executedensure proper functionality of each development team's code.

In another embodiment, controller 110 includes a set of test operationsto evaluate the provisioning of system resources to specific regionalcustomers. For example, controller 110 can include test operationsdesigned to provide the same software service from host servers 120physically located in different geographic areas. Controller 110 can usethe results such a test operation to identify optimal host severs 120for specific software subsystems. Controller 110, therefore, allows eachdevelopment team to independently balance the additional processingdemands imposed by the test operations against the likelihood of a codedefect being present.

FIG. 5 illustrates an example computing system for performing automatedtesting of software modules deployed on one or more host servers toidentify performance degradation in software modules and the source codecommit(s) that likely caused the degraded performance on the one or morehost servers. As shown, the controller 500 includes, without limitation,a central processing unit (CPU) 502, one or more I/O device interfaces504 which may allow for the connection of various I/O devices 514 (e.g.keyboards, displays, mouse devices, pen inputs, etc.) to controller 500,network interface 506, a memory 508, storage 510, and an interconnect512.

CPU 502 may retrieve and execute programming instructions stored in thememory 508. Similarly, the CPU 502 may retrieve and store applicationdata residing in memory 508. The interconnect 512 transmits programminginstructions and application data, among the CPU 502, I/O deviceinterface 510, network interface 506, memory 508, and storage 510. CPU502 is included to be representative of a single CPU, multiple CPUs, asingle CPU having multiple processing cores, and the like. Additionally,the memory 508 is included to be representative of a random accessmemory. Furthermore, the storage 510 may be a disk drive, solid statedrive, or a collection of storage devices distributed across multiplestorage systems. Although shown as a single unit, the storage 510 may bea combination of fixed and/or removable storage devices, such as fixeddisc drives, removable memory cards or optical storage, network attachedstorage (NAS), or a storage area-network (SAN).

As shown, memory 508 includes a test operation manager 545 configured toretrieve and execute test operations from test operation library 550 instorage 510. Test operation manager 545 communicates with host servers120 via network interface 520 to initialize each test operation and toreceive the output data set generated from each test operation. Whentest operation manager 545 receives an output data set from a hostserver 120, test operation manager 545 stores the output data set inoutput data set library 560, and in database 140 via network interface520. Output data set library 560 acts as local storage for output datasets used by defect manager to generate visualizations, detect defectsin the production code, and identify the root cause of a defect. Defectmanager 555 accesses output data sets from output data set library 560,database 140, or both, to score individual test operations and analyze acurrent test operation to identify defects. Defect manager 555 appliesstatistical analysis to the current output data set based on similarprior output data sets to identify deviations in software performancethat indicate a defect has been introduced into the code. Once a defecthas been detected defect manager 555 uses the context data of thecurrent output data set to retrieve software updates checked into sourcecode repository 130 preceding the execution of the current testoperation. Defect manager 555 compares the one or more software updatesto the software subsystems experiencing the defect, and associates thedefect with the software update whose scope most closely matches theperformance defect. After associating the defect with a software update,defect manager 555 notifies the development team responsible about thedefect and the associated software update.

Note, descriptions of embodiments of the present disclosure arepresented above for purposes of illustration, but embodiments of thepresent disclosure are not intended to be limited to any of thedisclosed embodiments. Many modifications and variations will beapparent to those of ordinary skill in the art without departing fromthe scope and spirit of the described embodiments. The terminology usedherein was chosen to best explain the principles of the embodiments, thepractical application or technical improvement over technologies foundin the marketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

In the preceding, reference is made to embodiments presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described embodiments. Instead, any combination of thepreceding features and elements, whether related to differentembodiments or not, is contemplated to implement and practicecontemplated embodiments. Furthermore, although embodiments disclosedherein may achieve advantages over other possible solutions or over theprior art, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the scope of the present disclosure. Thus,the aspects, features, embodiments and advantages discussed herein aremerely illustrative and are not considered elements or limitations ofthe appended claims except where explicitly recited in a claim(s).Likewise, reference to “the invention” shall not be construed as ageneralization of any inventive subject matter disclosed herein andshall not be considered to be an element or limitation of the appendedclaims except where explicitly recited in a claim(s).

Aspects of the present disclosure may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the present disclosure may take the form of a computerprogram product embodied in one or more computer readable medium(s)having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples a computer readable storage medium include: anelectrical connection having one or more wires, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.In the current context, a computer readable storage medium may be anytangible medium that can contain, or store a program.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A method for detecting and sourcing softwaredefects comprising: receiving an output data set from a host serverexecuting a test operation in a software system; comparing the outputdata set to a performance model; identifying that a defect exists insoftware executing on the host server based, at least in part, ondetermining that the output data set deviates from the performance modelby more than a threshold; retrieving a source code update from a sourcecode repository, wherein the source code update was committed to thesource code repository before the execution of the test operation;comparing a scope of the source code update to the defect; and notifyinga development resource related to the source code update of the defectupon determining that the scope of the source code update and the defectmatch.
 2. The method of claim 1, further comprising: generating avisualization including the performance model and the output data set;and displaying the visualization to a user.
 3. The method of claim 2,wherein generating the visualization comprises generating performancedata graphs for a plurality of subsystems of the software system,wherein each performance data graph illustrates the performance modeland the output data set for a specific subsystem of the plurality ofsubsystems.
 4. The method of claim 1, wherein the performance model isbased, at least in part, on performance data captured in a plurality ofprior output data sets.
 5. The method of claim 1, wherein the outputdata comprises a score weighing performance of each of a plurality ofsubsystems of the software system, and wherein identifying that thedefect exists comprises determining that the score differs from athreshold score defined by the performance model by a threshold amount.6. The method of claim 1, further comprising: identifying a secondsource code update deployed prior to a commit date of the source codeupdate; and deploying the second source code update to the host server.7. The method of claim 1, wherein the output data set includes metadatadescribing a software version that the test operation was executed in,metadata defining operations performed as part of the test operation,and metadata recording a response time for each operation.
 8. A system,comprising: a processor; and memory storing instructions which, whenexecuted on the processor, perform an operation for detecting andsourcing software defects, the operation comprising: receiving an outputdata set from a host server executing a test operation in a softwaresystem; comparing the output data set to a performance model;identifying that a defect exists in software executing on the hostserver based, at least in part, on determining that the output data setdeviates from the performance model by more than a threshold; retrievinga source code update from a source code repository, wherein the sourcecode update was committed to the source code repository before theexecution of the test operation; comparing a scope of the source codeupdate to the defect; and notifying a development team related to thesource code update of the defect upon determining that the scope of thesource code update and the defect match.
 9. The system of claim 8, theoperation further comprising: generating a visualization including theperformance model and the output data set; and displaying thevisualization to a user.
 10. The system of claim 9, wherein generatingthe visualization comprises generating performance data graphs for aplurality of subsystems of the software system, wherein each performancedata graph illustrates the performance model and the output data set fora specific subsystem of the plurality of subsystems.
 11. The system ofclaim 8, wherein the performance model is based, at least in part, onperformance data captured in a plurality of prior output data sets. 12.The system of claim 8, wherein the output data comprises a scoreweighing performance of each of a plurality of subsystems of thesoftware system, and wherein identifying that the defect existscomprises determining that the score differs from a threshold scoredefined by the performance model by a threshold amount.
 13. The systemof claim 8, further comprising: identifying a second source code updatedeployed prior to a commit date of the source code update; and deployingthe second source code update to the host server.
 14. The system ofclaim 8, wherein the output data set includes metadata describing asoftware version that the test operation was executed in, metadatadefining operations performed as part of the test operation, andmetadata recording a response time for each operation.
 15. Anon-transitory computer-readable medium comprising instructions which,when executed by one or more processors, performs an operation fordetecting and sourcing software defects, the operation comprising:receiving an output data set from a host server executing a testoperation in a software system; comparing the output data set to aperformance model; identifying that a defect exists in softwareexecuting on the host server based, at least in part, on determiningthat the output data set deviates from the performance model by morethan a threshold; retrieving a source code update from a source coderepository, wherein the source code update was committed to the sourcecode repository before the execution of the test operation; comparing ascope of the source code update to the defect; and notifying adevelopment team related to the source code update of the defect upondetermining that the scope of the source code update and the defectmatch.
 16. The non-transitory computer-readable medium of claim 15,wherein the operation further comprises: generating a visualizationincluding the performance model and the output data set; and displayingthe visualization to a user.
 17. The non-transitory computer-readablemedium of claim 16, wherein generating the visualization comprisesgenerating performance data graphs for a plurality of subsystems of thesoftware system, wherein each performance data graph illustrates theperformance model and the output data set for a specific subsystem ofthe plurality of subsystems.
 18. The non-transitory computer-readablemedium of claim 15, wherein the performance model is based, at least inpart, on performance data captured in a plurality of prior output datasets.
 19. The non-transitory computer-readable medium of claim 15, theoutput data comprises a score weighing performance of each of aplurality of subsystems of the software system, and wherein identifyingthat the defect exists comprises determining that the score differs froma threshold score defined by the performance model by a thresholdamount.
 20. The non-transitory computer-readable medium of claim 15further comprising: identifying a second source code update deployedprior to a commit date of the source code update; and deploying thesecond source code update to the host server.